Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

In recent years, even though Stochastic Gradient Descent (SGD) and its variants are well-known for training neural networks, it suffers from limitations such as the lack of theoretical guarantees, vanishing gradients, excessive sensitivity to input. To overcome these drawbacks, alternating minimization methods have attracted fast-increasing attention recently. As an emerging open domain, however, several new challenges need be addressed, including 1) Convergence properties sensitive penalty parameters, 2) Slow convergence rate. We, therefore, propose a novel monotonous Deep Learning Alternating Minimization (mDLAM) algorithm deal with two challenges. Our innovative inequality-constrained formulation infinitely approximates original problem non-convex equality constraints, enabling our proof proposed mDLAM regardless choice hyperparameters. is shown achieve fast linear by Nesterov acceleration technique. Extensive experiments on multiple benchmark datasets demonstrate convergence, effectiveness, efficiency algorithm.